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IMAGE CODING BY USE OF DISCRETE COSINE TRANSFORMS 

RELATED APPLICATIONS 
This application is related to U.S. Pat. application Serial No. 761,660 
filed Sep. 18, 1991, now U.S. Pat. No. 5,168,375. 

FIELD OF THE INVENTION 
This invention relates to the coding of image data by use of discrete 
cosine transforms and, more particularly, to coding which suppresses or reduces 
blocking effects even at relatively high ratios of image data compression. 

BACKGROUND OF THE INVENTION 
Image Processing and Data Compression 

Image processing is employed in various applications such as the 
electronic conversion of photographic images, the reproduction of graphical 
information in a printing operation, and the coding and subsequent reconstruction 
of digital image data in electronic communication systems. 

In such applications it is normally required to handle extremely large 
amounts of data in encoding and reproduction operations, especially color 
images. As a result, it becomes necessary to use high rates of data transfer if the 
rate of image conversion and reconstruction is to be kept within acceptable 
bounds. Consequently, there continues to be much effort expended in finding 
methods to decrease the amount of data required to effect the processing desired 
without compromising the quality of the image transmitted or reproduced. 

Image processing procedures can be conducted directly upon the picture 
data elements, or pixels. One of the most common image operations performed 
on images in the spatial domain, where pixel parameter value is specified as a 
function of its position with respect to other pixels, is convolution. Convolution 
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operations are particularly common for smoothing boundaries between adjacent 
pixels or groups of pixels so that, once processed, the likelihood of the presence 
of visual discontinuities over the image is reduced. However, the penalty is an 
increase in computational intensity. In spite of this computational overhead, 
5 there remain circumstances where convolution procedures are essential. 

Before a convolution operation is initiated, the numerical values 
representing image data are formatted into a matrix in which each of the matrix 
terms represents and describes a pixel from the original image. The particular 
type of processing, or filtering, to be performed on the image data is determined 

10 by the particular numerical values used as filter terms in the convolution 

operation. These filter terms are provided in a second matrix commonly denoted 
as the kernel matrix. 

The filtering characteristic of a kernel matrix can be more precisely 
specified as the number of terms used for a kernel matrix is increased. 

15 However, implementation of the convolution procedure becomes more 

complicated and requires a significantly greater processing capability with 
correspondingly more processing time with a larger kernel matrix. Because 
convolution is performed on numerical terms representing pixels (/. e. , image 
data), the operation is commonly designated as being performed in the spatial 

20 domain. 

Fig. 1 shows the basic operating steps of a convolution performed in the 
spatial domain. This operation can be represented mathematically as follows: 

/V-l 

h{i) = Y, s 01 * k{i~j) 

The original image data, here shown as H x V matrix 10, is convolved at 
operation 20 with an f x f spatial kernel matrix 12 to yield processed image data 
25 14 in the spatial domain. To perform a convolution requires a series of 
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multiplications followed by a summing operation for each processed term 
obtained. It can be appreciated that the computational effort expended increases 
geometrically as the sizes of the component matrices increases. 

Processing Using Transform Methods 

Reduced computation time may be realized when image data processing is 
performed in the frequency domain rather than in the spatial domain. Image data 
terms are converted into frequency coefficients by a matrix multiplication of the 
image date matrix by a suitable transform matrix. This conversion is commonly 
accomplished by utilizing an orthogonal transform such as a Fourier transform 
(FT) or a discrete cosine transform (DCT). To facilitate processing, the 
frequency-domain operations are performed on individual portions of the image- 
data matrix, and the resulting data are recombined into one matrix when the 
operations have been accomplished. 

A commonly-used frequency-domain processing method, described in ISO 
Draft International Standard 10918-1, 13.3, adopted by the Joint Photographic 
Experts Group (JPEG), is shown in Fig. 1. To facilitate the sequence of 
processing operations to be undertaken in the frequency domain, H x V image- 
data matrix 10, containing pixel values s y x from the original spatial image, is 
initially segmented into a blocked image-data set 16. Set 16 is a P' x Q' array 
of N x N image-data subset matrices 18, denoted by t e k where 

0 < x < (H - 1) and 0 < y < (V - 1), 
0 < k < (P' - 1) and 0 < t < (Q f - 1). 
P' = H/N, when H is an integral multiple of N 
Q' = v/N, when V is an integral multiple of N 

This segmentation of the image-data matrix is, in effect, a one-to-one mapping of 
each pixel value s v x from image-data matrix 10 to one of the P' x Q' array of 
N x N image-data subset matrices t . k 18. 
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Each image-data subset matrix 18 obtained by this segmentation operation 
is then processed as if it were a stand-alone matrix. Image-data subset terms s y> x 
are transformed, from image-data values in the spatial domain, into numerical 
values representing the image data in the frequency domain. The transform 
equation used as the DCT is actually for a discrete even cosine transform 
(DECT), hereinafter DCT, and is given by: 



for 0 < i < 7 and 0 < j < 7, 
where S(v,u) represents DCT coefficients; 

s(j,i) represents original image data points; 

s'(j,i) represents the reconstructed image data points; 

C tJ = — for u = 0; 

C u = 1 for u * 0; 

C v = _L for v = 0; and 

C v = 1 for v == 0; 

This is accomplished by transforming each image-data subset matrix t f k by 
means of the DCT. The matrix terms s y , are transformed in two dimensions in 
accordance with the following matrix operation given by: 




(2f + l)«7t 

16 



cos 



(2j+l)vn 
16 



i=0 y=0 



[S] = [D] [S] [D] T 



to obtain a P' x Q' array 22 of DCT coefficient matrices T f k 24. D is the 
DCT basis matrix. The terms S „ „ in DCT coefficient matrice.s 24 are usually 
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referred to as frequency coefficients. In most processing operations, these 
frequency coefficients can also be filtered at operation 26 by pointwise 
multiplication of each term S v u in DCT coefficient matrix T , k 24 with the 
reciprocal of the respective term Q v u of quantization table 28. The terms 
5 resulting from this pointwise multiplication, S v> u /Q V( u , comprise a P' x Q' 
array 30 of N x N DCT coefficient product matrices T' f k 32. 

After filtering, the matrix terms, S Vi u /Q Vi u , are usually "rounded off," or 
quantized, by a method well understood in the art. Quantized terms, denoted by 
Sq v u , replace the terms in DCT coefficient product matrices T' ft k 32 to yield an 

10 H x V quantized DCT coefficient matrix 34. Those terms which remain after 
quantization represent information which has been compressed from the original 
image data set. These DCT coefficient terms are then encoded at 36 and 
transmitted or stored. 

As the ratio of image compression is increased in such a process, an 

15 undesireable feature known as "blocking" becomes increasingly noticeable. This 
feature usually appears because the effect of data compression on a transformed 
block of image data differs slightly from the effect that the same process has on 
an adjacent block of .image data. As a result, artificial intensity discontinuities 
may appear along the boundaries of adjacent image data blocks in the 

20 reconstructed image. Thus, data compression as normally performed is affected 
by the undesirable effects of blocking. 

Two general approaches to reducing the blocking effect have been 
considered - as related in "Two-Dimensional Signal and Image Processing" (at 
p. 653) by Jae S. Lim, 1992. In one approach, the blocking effect is dealt with 

25 at the source. An example of this approach is the overlap method, which 
modifies the image segmentation process. A typical segmentation procedure 
divides an image into mutually exclusive regions. In the overlap method, the 
subimages are obtained with a slight overlap around the perimeter of each 
subimage. The pixels at the perimeter are coded in two or more regions. In 

30 reconstructing the image, a pixel that is coded more than once can be assigned an 
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intensity that is the average of the coded values. Thus, abrupt boundary 
discontinuities caused by coding are reduced because of an averaging effect 
tending to reduce differences between boundaries in reconstructed subimages. 
While the overlap method tends to reduce blocking effects, some pixels are 
5 coded more than once, and this increases the number of pixels coded. This 
increase shows why overlap of two or more pixels has not been considered very 
useful. In image coding, however, the cost of overlapping subimages is an 
increase in the number of computations and more seriously, a potential increase 
in the required bit rate if the overlapped, encoded image is transmitted. An 
10 overlap of more than one pixel has thus seldom been considered in DCT image 
coding. 

Quantization using the JPEC standard compression process causes 
blocking because there is no overlapping of image data and no "sharing" of data 
information between adjacent blocks of data. As a consequence, the problem of 
15 encountering blocking within the JPEG format continues to persist. 

Therefore, it is an object of the present invention to provide a method for 
achieving relatively high rates of data compression without incurring blocking 
artifacts. 

It is a further object of the present invention to provide a method for 
20 performing filtering without incurring blocking effects. 

It is a further object of the invention to provide a method for quantifying 
the factors responsible for the generation of blocking artifacts, such that the 
occurrence can be controllably suppressed or eliminated. 

Other objects of the invention will, in part, appear hereinafter and, in 
25 part, be apparent when the following detailed description is read in c nectron 
with the drawings. 
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SUMMARY OF THE INVENTION 
The present invention comprises a process and a system in which the 
processing steps take place in the frequency domain. The invention is based on 
the observation that (1) the method of convolving a spatial-domain kernel with an 
5 image-data matrix may be replaced by a pointwise multiplication of the DECT of 
subsets of the image-data matrix with the DOCT of the kernel if the image-data 
subsets overlap one another, and (2) the amount of overlap necessary to suppress 
the blocking effect is determined by the characteristic of the particular frequency 
matrix utilized. 

10 Derivation of these frequency-domain coefficients is accomplished by 

means of a discrete even cosine transfer (DECT) of the spatial-domain image 
data. Prior to transformation, the image data is reformatted, by means of a one- 
to-many mapping of each image datum into an array of image-data submatrices. 
This mapping operation, which produces terms common to more than one 

15 submatrix. is commonly referred to as overlapping" of the image data. The 

processing of frequency coefficients in the frequency domain is accomplished by 
means of a filter matrix or by a filter matrix which has been derived from a 
spatial-domain kernel matrix. The characteristics of the frequency matrix 
provide criteria for determining the amount of overlap to be used to avoid 

20 blocking artifacts. Processed coefficients in JPEG format may be transmitted or 
stored using standard JPEG hardware or its equivalent after unoverlapping. 
Subsequently, these coefficients can undergo the inverse sequence of operations 
by which the original image-data matrix was processed, to yield the original 
image data. 

25 The effects of blocking are suppressed for various levels of image data 

compression and compression levels of from 20:1 to 100:1 can be realized while 
maintaining similar or equivalent quality levels. 

The invention provides for a major saving in electronic circuitry by use of 
existing JPEG array transform circuitry modified to provide access to the DCT 

30 and the Inverse Discrete Cosine Transform (IDCT), or by use of an equivalent 
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digital signal processor. Therefore, in any electronic signal processing system 
which incorporates JPEG (or MPEG) compression equipment, the invention can 
be practiced by use of available array transform circuitry. Furthermore, since 
the JPEG cosine transformer operates at great speed, the signal processing of the 
invention can be accomplished rapidly. It is also possible to implement the 
invention by way of software adapted for use with general-purpose computers. 

BRIEF DESCRIPTION OF THE DRAWINGS 
The objects and features of the present invention will become more 
apparent from the consideration of the general description given above and the 
detailed description of the preferred embodiments given below, taken in 
conjunction with the accompanying drawings in which: 

Fig. 1 is a block diagram of a conventional method of processing image 

data. 

Fig. 2 is a block diagram illustrating the processing sequence performed 
on image data as taught by the present invention. 

Fig. 3 is a diagram illustrating the mapping of image data into an array of 
submatrices as part of the processing sequence of Fig. 2. 

Fig. 4 illustrates the mapping process of Fig 3 in which submatrices are 
derived. 

Fig. 5 is a diagram illustrating generation boundary submatrices 
comprising the array of Fig. 3. 

Fig. 6 illustrates the steps taken to format a mask multiply matrix from a 
kernel matrix as pan of the processing sequence of Fig. 2. 

Fig. 7 illustrates the steps taken to format a kernel matrix from a mask 
multiply matrix as part of the processing sequence of Fig. 2. 

Fig. 8 is a block diagram showing a method for determining overlap. 

Fig. 9 is a block diagram showing a method for regenerating spatial 
image data after the processing procedure shown i ; n Fig. 2. 
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Fig. 10 illustrates the steps taken to either unoverlap or to reoverlap a 
given matrix. 

Fig. 1 1 is a block diagram schematically showing an apparatus for 
practicing the claimed invention. 

DETAILED DESCRIPTION OF THE INVENTION 
General Overview of Procedure, Reference to Mathematical Equations 

Fig. 2 is an overview of the general steps taken to perform the image 
coding procedure of the invention. Image coding in the present invention is 
performed in a manner similar to the conventional method as described above. 
That is, the image-data matrix is processed in portions, to facilitate the coding 
procedure, and the results are then recombined to yield the desired information. 
The coding operation can be performed in more than one way: (i) the steps 
described below can be performed on a first portion of the image-data matrix 
before being performed on a second portion of the image-data matrix, (ii) one 
step can be performed on all portions of the image-data matrix before a 
subsequent step is initiated, or (iii) a combination of (i) and (ii) can be used. 

There is shown in Fig. 2 an H x V image-data matrix 50 containing 
image-data terms s - t r A mapping operation is used to generate a P x Q array 
52 of N x N image-data submatrices 54 denoted as b n m . By means of this 
operation, each image datum s Jti is multiply-mapped into more than one image- 
data submatrix 54. This process results in an array in which resulting 
submatrices 54 have terms in common with one another. Preferably, each 
submatrix 54 has at least one row or one column of data which is identical to a 
row or column of data present in an adjacent submatrix 54. 

Each submatrix 54 is then transformed to yield array 60 of N x N 
frequency -domain submatrices 62. each denoted as B n m . and each submatrix 62 
having terms denoted by S Ktl . These submatrix terms S „ M are derived from the 
original image data terms s, : by means of the DECT equation: 
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x) COS 



16 



cos 



(2/+l)V7t 

16 



10 



15 



20 



for 0 < \x < 7 and 0 < v < 7. This particular expression of the 
DECT equation, applicable to 8 x 8 submatrices, is intended for convenience of 
illustration only and is not meant to limit the application of the inventive method. 

The set of terms {s j( t } used in the deviation is determined by the 
submatrix 54 selected for the derivation. The terms S „ M include floating-point 
values as a result of the above matrix multiplication. These terms are then 
quantized at 62 as is well-known in the art. The resulting array is then 
unoverlapped at 64 by a process to be explained in further detail below, from 
which H x V matrix 66 of quantized frequency-domain coefficients is obtained. 

In a preferred embodiment, frequency-domain submatrices 62 are 
pointwise multiplied by mask matrix 72 at operation 74 before the operations of 
quantization or unoverlapping of array 60 are performed. The terms F v ^ of 
mask matrix 72 consist of frequency coefficients and may be provided in the 
form of mask matrix terms. Alternatively, a spatial-domain kernel matrix 12 
may be provided from which mask matrix terms F „ mlt can be derived. In the 
inventive method, the size of kernel matrix 12 is used at operation 76 to 
determine the amount of overlap necessary to suppress blocking effects. If a 
mask multiply operation has been performed, then the result is quantized and 
unoverlapped, as described above, to yield a matrix 78 of quantized filtered 
frequency-domain coefficients. 

These blocks of coefficients in either matrix 66 or matrix 78 may be 
further encoded for transmittal at 80. Before encoding, the matrix frequency- 
domain coefficients are grouped into appropriately-sized blocks (e.g., 8 x 8 for 
JPEG) according to the requirements of the processing standard utilized. 
Preferably, zig-zag sequencing at 82, well-known in the art, is performed on the 
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blocks of coefficients, and then the sequence is Huffman encoded at 84 prior to 
transmittal, as is also well-known in the art. 

After transmittal, the H x V image-data matrix 50 of Fig. 2 can be 
recovered by performing the above sequence in reverse, shown in Fig. 9. A 
5 received file, such as JPEG-compatible file 340 is Huffman decoded at block 342 
and the resulting data stream is reverse zig-zagged at block 344. The resulting 
frequency coefficients are transformed with a IDCT to produce spatial terms at 
block 346 which are then reoverlapped in accordance with the right-side 
sequence of operations shown in Fig. 10. These spatial terms are then 

10 transformed back into frequency coefficient terms 314, also denoted as S'(j,i) in 
block 348 of Fig. 9. If a filtering operation had been performed on the image 
data, multiplication of terms S'(j,i) by inverse filter 352 is next performed at 
350. The resulting terms are then transformed back into spatial terms by 
applying an IDCT at 354 to produce overlapped set 52' of terms denoted by 

15 s'(j,i). These terms correspond to the terms of matrix 52 in Fig. 2. After an 
unoverlap has been performed at block 356 in accordance with the procedure 
shown in Fig. 10, spatial image data 50' is obtained, which corresponds to the 
original spatial image data contained in matrix 50 of Fig. 2. The following 
differences are noted between the processing of data for transmittal or storage as 

20 shown in Fig. 2, and the recovery process as shown in Fig. 9 and described 
above: 

a. re-overlapping matrix 78 or 66 at block 346 in Fig. 9 differs from 
the unoverlapping operation 64 in Fig. 2, as explained below; 

b. the filtering operation 74 in Fig. 2 performed to produce matrix 78 
25 is undone with multiplication at 350 by reciprocal filter 352 in 

Fig. 9; 

c. matrix 52 in Fig. 2 is obtained by multiplication with an inverse 
discrete even cosine transform (IDECT) at 354 in Fig. 9, given by 
the following expression: 
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*'m -^i^ 1 « ja iF 



H=0 v=0 

for 0 < i < 7 and 0 < j < 7, 
where S(^,/x) represents formatted DCT coefficients; 
s'(j,i) represents the processed image data; 

C, = J- for fi = 0; 
v/2" 

= 1 for m ^ 0; 
C, = -L for v = 0; and 

C = 1 for v * 0; 



DETAILED STEPS 
Mapping 

10 A more detailed description of the mapping process shown in Fig. 2 is 

provided in Fig. 3. Image-data submatrices 100 and 102, designated as b n m and 
t>ri. m+i respectively, have been generated from image-data matrix 50 as in Fig. 2. 
Each term s jt s of matrix 50 has been mapped into one or more of the P x Q 
image data submatrices 54, denoted by b n m , where 

15 0 < m < (P - 1), 0 < n < (Q - 1). 

As stated above, the entire mapping operation need not be completed before 
subsequent processing steps can begin on those submatrices already generated. 

In Fig. 3, submatrix 100 is adjacent to submatrix 102 within row "n M of P 
x Q array 52 of Fig. 2. The set of rerms {x„^} in submatrix b n m , where 0 < 

20 fi.v < (N-l), corresponds to the image-data matrix set of terms {sj } within 

square region 104. shown in solid lines, and the term set {x„ J in submatrix b n 
m _t corresponds to image-data matrix term set h within square region 106, 
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shown in long dashed lines. Image-data terms s j< ; lying within area 108 in 
which the two square regions overlap one another are mapped into both 
submatrices 100 and 102. It can be seen that the terms in column 110 of 
submatrix 100 are the same as the terms in column 112 of submatrix 102. The 
5 mapping process generates submatrices having at least one column in common 
with an adjacent submatrix lying in the same row of P x Q array 52. 

A similar relationship exists between adjacent submatrices lying in the 
same column of P x Q array 52. The term set {x„ for submatrix 120, 
designated as b n+l m corresponds to image-data matrix term set {s } } within 

10 square region 122, shown in short dashed lines. Image-data terms s } ti lying 
within area 124 in which square region 122 overlaps square region 104 are 
mapped into both submatrices b n+1 m 120 and b n m 100. It can be seen that 
submatrices 120 and 100 will have one row or more containing terms common to 
each submatrix. The mapping process thus also generates submatrices having at 

15 least one row in common with an adjacent submatrix lying in the same column of 
P x Q array 52. 

The set of terms {x^ „} of any image-data submatrix b n m can be derived 
from the image-data set {sj } by means of a stepping-increjnent integer A. This 
integer A corresponds to the number of rows or columns to be "skipped," after 
20 one square region has been used to derive terms for one submatrix, and before 
the next square region is defined from which image-data matrix terms are 
mapped into an adjacent submatrix. From Fig. 3, it can be seen that A is related 
to the amount of overlap by the expression 

A = (N - overlap). 

25 The set of terms {x „ ,} of the image-data submatrix in the p-th column and the 
q-th row of said P x Q array are derived from the terms s J j in the image-data 
set according to the generating equation: 

= s,.^ u ^ where 0<p<(P-l) and 0<q<(Q-l). 
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There is shown in Fig. 4 four adjacent submatrices: 130, 132, 134, and 
136 designated as b 00 , b 0i! , b r(h and b u respectively. The terms x ^ within 
these submatrices were mapped from image-data matrix 50 in accordance with 
the equation above. In a preferred embodiment, the size of the generated 
5 submatrices is 8-pixels by 8-pixels, a format which is compatible with hardware 
meeting the JPEG standard. In the example shown, adjacent submatrices have 
been overlapped by four rows or four columns of image data. Submatrices b 0 0 
and b 0 { have four columns containing common terms - those columns headed by 
terms s 0 4 through s 0i 7. 

10 The image-data submatrices in the next row of the P x Q array are 

derived in the same manner as the submatrices in the previous row. Since the 
overlap process operates in the vertical direction as well as the horizontal, the 
upper rows of each submatrix have terms x ViX in common with the bottom four 
rows of the submatrices in the first row of the array. In the example shown, 

15 submatrices b 00 and b 10 overlap by four rows. Four adjacent rows in the two 
submatrices contain common terms - those rows beginning with the terms s 4 0 
through s 7 0 . This process is continued until all possible submatrices have been 
generated by means of the generating equation. 

However, as can be seen, not all the image-data matrix terms have been 

20 mapped into four submatrices. Image-data term s 0 0 , for example, is mapped 
only into submatrix b 0 0 , and image -data term s 0 4 is mapped only into 
submatrices b 0 0 and b 0 . i- In a preferred embodiment, additional submatrices 
are generated by which all terms in the image-data matrix are mapped into 
multiple submatrices. These additional submatrices include the four "corner" 

25 submatrices of the array, and the other "border" submatrices Sying between the 
four "corner" submatrices. 

In the example at hand, these "corner" and "border" submamces of the P 
x Q array, are formed as shown in Fig. 5. A "comer" submatrix 140, denoted 
as b . ; .,, is formed by mapping the sixteen image-data matrix terms s 0 0 through 

30 s 0 3 . s 1 j, through s , 3 , s : {/ through s : _ ? . and s 3 0 through s 3 3 into the s:ame 
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relative positions of the rightmost lower quadrant 142 of submatrix b., _,. The 
other three quadrants 144, 146, and 148 each contain the same sixteen image- 
data terms, mapped in such a relationship that submatrix b.^., is symmetrical 
both across its vertical centerline 150 and across its horizontal centerline 152. 
5 The other three "corner" matrices are formed in a similar manner - by multiply 
mapping image-data matrix terms from the respective corners of image-data 
matrix 50. ' 

A "row-border" submatrix 160, designated as b.i 0 , is formed by mapping 
the first eight image-data terms from each of the top four rows of matrix 50 — 

10 that is, s 0 o through s 0> 7* s 1, 0 through s t 7 , s 2i 0 through s 2 , 7* a *id s 3, 0 through s 
3 7 — into the same four relative positions of lower half 162 of submatrix b_ li0 - 
The terms for top half 164 of submatrix b., 0 are these same rows placed in 
reverse sequence such that submatrix b. t 0 is symmetrical about its horizontal 
centerline 166. That is, the first and eighth rows are identical, as are the second 

15 and seventh, the third and sixth, and the fourth and fifth. The remaining "row- 
border" submatrices in array 52 are derived in a similar manner. 

A "column-border" submatrix 170, designated as b 0 . } , is formed by 
mapping thirty two image-data matrix terms - eight each from the the columns 
beginning with terms s 0 0 , s 0 { , s 0 . 2< an ^ s o. 3 ~~ i nt ° the same relative positions 

20 in the right half 172 of submatrix b 0 The terms for left half 174 of submatrix 
b 0 .[ are these same rows placed in reverse sequence such that submatrix b 0 is 
symmetrical about its vertical centerline 176. The remaining "column-border" 
submatrices in array 52 are derived in a similar manner. 

It should be noted that, in ine aspect of the inventive procedure, every 

25 image datum Sj . has been mapped into four different adjacent submatrices by 
means of the overlapping operation performed prior to transformation into the 
frequency domain. When image-data terms s , : are thus repeated, or overlapped, 
between adjoining submatrices, the likelihood of blocking effects is lessened as 
discussed above. This overlapping is necessary because subsequent processing is 
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performed sequentially, on individual submatrices, rather than being performed 
on the entire image-data set as a whole. 

Transformation into Frequency Coefficients 

Each term x „ M of the image-data submatrices is next transformed into a 
5 frequency-domain submatrix term X 9t „ by applying the DECT: 



X(v,^) = -C C v Jj J> ^0,0 cos v cosl ^77 — 

4 P I=0 ;=0 16 1& 



for 0 < ii < 7 and 0 < i> < 7 if submatrix size has been selected as 
8x8. 

This transformation is accomplished by performing the following matrix 
multiplication with each of the image-data submatrices: 
10 [B,J = [D] [b,.J [D T ] 

where D is the DECT basis matrix. 

Derivation of the Mask Matrix from the Spatial Kernel 

Derivation of mask matrix 72, of Fig. 2, from spatial kernel matrix 12 is 
illustrated in Fig. 6. The terms contained in the lower righthand quadrant 180 of 
15 spatial kernel 12 are selected from which an N x N matrix may be formed. If 
kernel matrix 12 is an odd symmetrical matrix, the center term 182 is included in 
quadrant 180. 

If quadrant 180 is N x N in size, the rerms k j ; of quadrant 180 are 
transformed into frequency terms F „ ^ by means of the discrete odd cosine 
20 transform (DOCT). 

A discrete odd cosine transform (DOCT) is applied to the matrix thus 
obtained, where 
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F(n,m) = 4 £ d i d k p cos-^ cos 



N-i Af-1 




7V71 

1 M 



for 



0 < m < N-l and 0 < n < N-l, 



where 



k P (j,i) is the two dimensional padded kernel matrix; 

N is the number of elements of k P (j,i) in the first dimension; 

M is the number of elements of k P (j,i) in the second dimension; 



5 



= Vi for i = 0; 



d ; = 



1 for i = 1, 2, 



(N-l); 



10 



dj = Vz forj = 0; 

d. = i forj = 1, 2, ...(M-l); 

i, j, fij v, m, n are integers; and 

kp(j,i) = 0 for | i | or | j | > (k-l)/2 and k is the kernel size. 



If quadrant 180 is larger than an N x N matrix, the extra row(s) and 
column(s) 184 are discarded, retaining term 182 as shown to produce N x N 
matrix 190. If quadrant 180 is smaller than an N x N matrix, additional row(s) 
and column(s) 186 are padded with zeroes to produce N x N matrix 192. Either 
15 matrix 190 or matrix 192 is then transformed, as above, by means of the DOCT 
to produce mask matrix 72: 



Derivation of the Spatial Kernel from the Filter Matrix 

It is usually more convenient to design a filter by working with frequency 
20 coefficients, in the frequency domain, than it is to work with kernel values in the 
spatial domain. But. since determination of an overlap value requires evaluation 
of the kernel matrix, it is necessary to derive an equivalent kernel matrix when a 



[F] = [D 0 ] [k] [D 0 ] T 
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10 



t r 

ID 



mask multiply filter matrix is provided. Fig. 7 illustrates the steps taken to 
format a kernel matrix from a mask multiply, or filter, matrix 2& r 0. Filter matrix 
280 is converted into an equivalent kernel matrix 282 by means of an IDOCT. If 



performed to yield an N x N matrix. This is accomplished by deleting one or 
more rows and columns from kernel matrix 282 to produce a truncated kernel 
matrix 12a. If the resulting kernel matrix 282 is smaller than an N x N matrix, 
a padding operation is performed. This is accomplished by adding one or more 
rows and columns of zeros to kernel matrix 282 to produce a padded kernel 
matrix 12b. In either case, the amount of overlap required to perform image 
coding without the occurrence of the blocking effect is determined by evaluation 
of equivalent kernel matrix 282. 

Referring now to Fig, 8, this procedure is as follows: 
The frequency-coefficient terms in filter matrix F* in block 250 are 
converted to spatial-domain values K" by multiplication as follows: 



for 0 < fi < N-l and 0 < v < N-l, 

where h P (j,i) is the two dimensional padded kernel matrix; 

N is the number of elements of h P (j,i) in the first dimension; 

M is the number of elemenfs of h P (j,i) in the second dimension; 

di = 1 for i = 0; 

dj - 2 for \ = 1. 2, ... (N-l); 



the resulting kernel matrix 282 is larger than an N x N matrix, a truncation is 



[IC] = [DoT [F - ] [D 0 ] 
where D 0 , which is a DOCT, is defined as follows: 



i-t 



N-\ M-l 
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dj = 1 forj = 0; 

dj = 2 forj = l t 2, ...(M-l); 

i, j, m, n, N, M are integers; and 

kp(j»0 = 0 for | i | or | j | > (k-l)/2 and k is the kernel size 

5 This yields equivalent kernel K" in block 252. Next, the equivalent kernel, K\ 
is truncated to yield a derived kernel matrix as shown in block 254. A DOCT is 
performed on the derived kernel matrix to return to frequency space and yields a 
modified filter matrix shown in 256. The RMS between the modified filter 
matrix of block 256 and the starting filter matrix, F\ in block 250 are then 
10 compared to quantify their differences as indicated in block 258. The RMS 

difference is then compared with a predetermined design figure of merit, FOM, 
in block 260 and, if the difference is sufficiently small, the size of the kernel, k, 
has been established. The overlap is then given by the quantity (k-1). 

In assessing the nature of the difference between the truncated equivalent 
15 spatial kernel, and its retransformed modified kernel matrix, several alterations 
may be made by performing the necessary forward and reverse discrete odd 
cosine transforms and empirical studies of the. quality of the particular image may 
and/ or other figures of merit may be considered. 

Mask Multiplication 

Each of the frequency-domain submatrices 62 in the resulting array 60 is 
then mask multiplied by frequency-domain mask matrix 72, or filtered. This is 
done by a pointwise multiplication of each term S }i in frequency-domain 
submatrix 62 with its corresponding term K } . from frequency-domain mask 
matrix 72 to yield terms S'j , where 

S',, = S K: x K, r 
Input matrix array 300, comprised of frequency terms in 8 x 8 matrices, 
for example, is transformed by an IDCT operation at 302 in Fig. 10 into array 
304 of 8 x 8 matrices containing spatial image terms. Because of the mapping 



20 



25 
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performed earlier in the sequence of operations, the matrices in array 304 
includes redundant or incorrect terms. These are removed by deleting the outer 
rows (4 total in this example) and outer columns (4 total) of each matrix in array 
304 to produce array 306 containing 4x4 matrices. Array 306 is then 
5 transformed using the DCT at 308 to yield matrix 310, which is then compatible 
with a processing standard, such as the JPEG standard. 

The reoverlapping of matrix 300 is accomplished first by transforming by 
using the IDCT at 302 to produce array 304' of 8 x 8 matrices containing 
spatial image terms. Array 304' is mapped into array 312 of 8 x 8 matrices, 
10 using the same mapping procedure described above. Array 312 is then 
transformed at 308, using the DCT, into array 314 of frequency terms. 

Hardware Disclosure 

Fig. 11 shows a system 200 for presenting an image of a scene 202 upon 
a display 204. Display 204 may be a printer which prints out the imaging data 

15 point by point and, preferably, provides a gray-scale or color representation of 
each pixel. Alternatively, display 204 may be a cathode ray tube (CRT), liquid 
crystal display (LCD), or light-emitting diode (LED) display. Image data of 
scene 202 is gathered by means of an optical scanner 206 which comprises a 
scanning mirror 208, a lens 210, a scan controller 212, a charge-coupled device 

20 (CCD) array detector 214, a sampler 216, and a storage unit 218. Scanner 206 
is provided by way of example for acquiring image data by the scanning of either 
a single line array or a two-dimensional array of photodetectors, where the image 
data may be obtained in various ways, including the scanning of a photographic 
slide of scene 202, or from an image produced by a television videocon. 

25 In operation, individual elements or cells of detector 214 detect optica\ 

radiation emanating from scene 202. Controller 212 drives mirror 208 in a 
pivoting movement about an axis 220 whereby rays 222 of the scene radiation 
are reflected through lens 210 by which the radiation is focused upon a face of 
detector 214. As mirror 208 pivots, scene 202 is scanned, line-by-line, onto 



WO 95/15530 



PCT/US94/13787 



detector 214 where each cell 224 of detector 214 views a scanned line of scene 
202. Each cell 224 outputs an analog electric signal in response to detection of 
incident radiation, where the amplitude of the signal increases with increased 
intensity of the incident radiation. Cells 224 are arranged in a single line along 
5 the face of detector 214 in scanner 206, the line being perpendicular to the plane 
of scanning. If desired, optical scanner 206 can be replaced by a CCD camera 
(not shown). Because CCD cells 224 in a camera are arranged in a two- 
dimensional array of rows and columns, an image of scene 202 could be obtained 
without requiring a scanning mirror 208. 

10 Sampler 216 is driven by controller 212 in synchronism with the scanning 

movement of mirror 208 to obtain the signals outputted by detector cells 224. 
Sampler 216 includes an analog-to-digital converter 226 which converts the 
analog signals of detector cells 224 to digitized data samples of an image of 
scene 202. The digitized samples are outputted by sampler 216 to storage unit 

15 218 for storage therein. 

As discussed above, the image data can be compressed prior to the 
displaying of the data on display 204. This may be done to facilitate data storage 
or data transmission, or both. Data storage in a storage unit 228 'may comprise a 
magnetic medium, an optical disk, or may be in the form of a solid-state read- 

20 only memory. Additionally, any of these methods can be configured into a card 
for ease in handling by a user. In this way, image data which has conventionally 
been outputted as a photograph may be stored on the card in accordance with the 
invention. 

The digitized samples outputted from storage unit 218 are compressed by 
25 data compression section 230. The invention can also be practiced with a 
transmission link 232. indicated in phantom, if it is desired to electronically 
transmit data from scanner 206 to display 204. In such an application, 
compression of the image data facilitates the electronic transmission of the data. 
Transmission link 232 can be a telephone circuit, a satellite communication link, 
30 or any other suitable form of communication system. 
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What is claimed is: 

1. A method of processing an image signal, including filtering by means 
of a spatial-domain kernel, comprising the steps of: 

acquiring the image signal, said image signal formatted as a series of 
electrical signals corresponding to image element characteristics; 
5 transforming said series of electrical signals into a set of image data, said 

set of image data formatted as an H x V matrix, having terms denoted by s jt if 
where 0 < i < (H-l) and 0 < j < (V-l); 

generating aPxQ array of N x N image-data submatrices from said set 
of image data, where N < H, N < V, P > int(H/N) and Q > int(V/N), each 
10 said image-data submatrix having terms denoted by x „ where 0 < v < (N- 
1), wherein each said image datum is mapped into at least one of said image-data 
submatrices, and further wherein each said image-data submatrix has at least one 
row or one column identical to a row or column of an adjacent image-data 
submatrix; 

15 performing a DECT on each said image-data submatrix to obtain a P x Q 

array of N x N frequency-domain submatrices, each said frequency domain 
submatrix having terms denoted by X, ^; 

applying the spatial-domain kernel, said kernel formatted as a symmetrical 
f x f matrix, where f < N; 

20 constructing an N x N kernel matrix from said spatial-domain kernel, 

with terms denoted by k„ <w , wherein each kernel matrix term is either assigned a 
value equal to a term in said spatial-domain kernel or else is set to zero; 

performing a DOCT on said kernel matrix to obtain an N x N frequency- 
domain mask matrix, having terms denoted by F„ tU \ 

25 filtering said frequency-domain submatrix by performing a pointwise 

multiplication of each said frequency-domain submatrix with said frequency- 
domain mask matrix to yield a P x Q array of N x N filtered frequency-domain 
matrices, havine terms denoted bv X'„ „ wherein 
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performing a quantization operation on the terms in said filtered 
frequency-domain matrices to obtain a P x Q array of N x N quantized filtered 
frequency-domain matrices; and 
5 unoverlapping said quantized filtered frequency-domain matrices to obtain 

a P x Q array of N' x N' quantized filtered frequency-coefficient matrices, 
where N' < N, said quantized filtered frequency-coefficient matrices comprising 
a compressed set of filtered image data which is functionally equivalent to the 
DCT of the convolution of said set of image data with said spatial-domain kernel, 
10 wherein said compressed set of filtered image data can be stored using a digital 
signal processor or similar electronic circuitry, or encoded and transmitted by 
electronic means, whereby said compressed set of filtered image data can be 
subsequently electronically reprocessed to regenerate the image, corresponding to 
the original image signal, without incurring blocking artifacts. 

15 2. The method of claim 1 wherein said step of generating a P x Q array 

of N x N image-data submatrices further comprises the steps of: 

a. ) determining a stepping-increment integer A, where A < N; 

b. ) mapping each term of the image data set {s } { } into at least one of 
said image-data submatrices, the terms x , M of said image-data submatrix in the 

:0 p-th column and the q-th row of said P x Q array being defined by the 
expression 

= s,,^^^, where 0 < p < (P-l) and 0 < q < (Q-l). 

3. The method of claim 2 wherein said step of determining a stepping- 
increment A comprises the steps of: 
5 a.) selecting an overlap factor A. wherein A = (N - 0; 

b.) evaluating an error function, expressible as g(A), against a pre- 
established criterion; and 
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c.) setting A equal to (N - A) if said criterion is met, otherwise 
incrementing A by one and repeating said steps of evaluating an error function 
and setting A equal to (N - A) if said criterion is met otherwise incrementing A 
by one, until said criterion is met. 

5 4. The method of claim 1 wherein said step of unoverlapping said 

quantized filtered frequency-domain matrices comprises the steps of 

a.) applying an Inverse Even DCT to each said quantized filtered 
frequency-domain matrix to obtain aPxQ array of N x N inverse-even 
coefficient matrices; 
10 b.) applying a "matrix pick" operation to each said inverse-even 

coefficient matrix to obtain aPxQ array of N' x N' reduced matrices such 
that said reduced matrices do not contain redundant matrix terms found in said 
inverse-even coefficient matrices; and 

c.) applying a Forward DCT to each said reduced matrix to obtain a P x 
15 Q array of N' x N' quantized filtered frequency-coefficient matrices. 

5. A method of processing an image signal, including filtering by means 
of a set of frequency coefficients, comprising the steps of: 

acquiring the image signal, said image signal formatted as a series of 
electrical signals corresponding to image element characteristics; 
20 transforming said series of electrical signals into a set of image data, said 

set of image data formatted as an H x V matrix, having terms denoted by Sj 
where 0 < i < (H-l) and 0 < j < (V-l); 

acquiring the set of frequency coefficients, said set of frequency 
coefficients formatted as an N* x N' frequency-coefficient matrix; 
25 determining a stepping-incremem integer A as a function of N\ 

expressible as A(N'); 

generating a P x Q array of N x N image-data submatriccs, where P > 
int(H/N ) and Q > int(V/N) and where N < H and N < V, each s^aid image- 
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data submatrix having terms denoted by x M , where 0 < /x,i> < (N - 1), 
wherein each image datum is mapped into at least one of said image-data 
submatrices, and further wherein each said image-data submatrix has at least (N- 
A) row(s) or (N-A) column(s) identical to (N-A) row(s) or column(s) of an 
5 adjacent image-data submatrix; 

constructing an N x N frequency-domain mask matrix, from said 
frequency-coefficient matrix, having terms denoted by F„ <M , wherein each 
frequency-domain mask matrix term is either assigned a value equal to a term in 
said frequency-coefficient matrix or else is set to zero; 
10 performing a DECT on each said image-data submatrix to obtain a P x Q 

array of N x N frequency-domain submatrices, each said frequency domain 
submatrix having terms denoted by X „ t „; 

filtering said frequency-domain submatrix by performing a pointwise 
multiplication of each said frequency-domain submatrix with said frequency- 
15 domain mask matrix to yield a P x Q array of N x N filtered frequency-domain 
matrices, having terms denoted by X'„ tM wherein 

X' = F x X 

performing a quantization operation on the terms in said filtered 
frequency-domain matrices to obtain a P x Q array of N x N quantized filtered 

20 frequency-domain matrices: and 

unoverlapping said quantized filtered frequency-domain matrices to obtain 
a P x Q array of N' x N' quantized filtered frequency-coefficient matrices, 
where N' < N, said quantized filtered frequency-coefficient matrices comprising 
a compressed set of filtered image data which can be stored using a digital signal 

25 processor or similar electronic circuitry, or encoded and transmitted by electronic 
means, whereby said compressed set of filtered image data can be subsequently 
electronically reprocessed to regenerate the image, corresponding to the original 
image signal, without incurring blocking artifacts. 
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6. The method of claim 5 wherein said step of generating a P x Q array 
of N x N image-data submatrices further comprises the step of deriving the 
terms x ^ v of the image-data submatrix in the p-th column and Hie q-th row of 
said P x Q array in accordance with the expression 

5 X,., = s v+q ^ +p . where 0 < p < (P-1) and 0 < q < (Q-1). 

7. The method of claim 5 wherein said step of determining a stepping- 
increment integer A comprises the steps of: 

a. ) performing a matrix multiplication of said frequency -coefficient 
matrix with an inverse DOCT matrix to obtain an N" x N" modified frequency 

10 coefficient matrix; 

b. ) performing a matrix multiplication of said modified frequency- 
coefficient matrix with a transformed inverse DOCT to obtain an N* x N" 
equivalent kernel matrix; 

c. ) truncating said equivalent kernel matrix to obtain an f x f derived 
15 kernel matrix, where f < N*; and 

d. ) selecting stepping-increment A to be given by (N - f + 1). 

8. The method of claim 7 wherein said step of determining a stepping- 
increment integer A further comprises the steps of: 

a.) selecting an overlap factor A to be given by (N - A); 
20 b.) evaluating an error function, expressible as g(A), against a pre- 

established criterion; and 

c.) setting A equal to (N - A) if said criterion is met, otherwise 
incrementing A by one and repeating said steps of evaluating said error function 
and setting A equal to (N - A) if said criterion is met otherwise incrementing A 
25 by one, until said criterion is met. 

9. The method of claim 5 wherein said step of unoverlapping said 
quantized filtered frequency-domain matrices comprises the steps of 
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a. ) applying an Inverse Even DCT to each said quantized filtered 
frequency-domain matrix to obtain a P x Q array of N x N inverse-even 
coefficient matrices having terms denoted by 0, M ; 

b. ) applying a "matrix pick" operation to each said inverse-even 

5 coefficient matrix to obtain a P x Q array of N' x N' reduced matrices such 
that reduced matrices do not contain redundant matrix terms found in said 
inverse-even coefficient matrices; and 

c. ) applying a Forward DCT to each said reduced matrix to obtain a P x 
Q array of N' x N' filtered frequency-coefficient matrices. 

10 10. The method of claim 9 wherein said step of applying a "matrix pick" 

operation further comprises the steps of: 

a. ) comparing each term 9 Vi M to at least one predetermined criterion; and 

b. ) mapping those terms meeting said at last one predetermined criterion 
into said N' x N'* reduced matrix. 
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